Skip to content

Expose cooperative generation and Core ML placement - #461

Merged
ivan-digital merged 1 commit into
mainfrom
perf/inference-resource-controls
Aug 22, 2026
Merged

Expose cooperative generation and Core ML placement#461
ivan-digital merged 1 commit into
mainfrom
perf/inference-resource-controls

Conversation

@ivan-digital

Copy link
Copy Markdown
Member

Summary

  • add token-boundary cooperative cancellation to Gemma 4 streaming generation
  • let Nemotron callers select Core ML compute units while preserving .all as the default
  • add opt-in real-model gates for cancellation and CPU+ANE output parity

Validation

  • curated unit suite: 233 passed, 6 skipped
  • Gemma cancellation E2E: passed with the cached E4B int4 checkpoint
  • Nemotron placement E2E: exact output parity; .all 1465.81 ms, CPU+ANE 1071.30 ms
  • release build completed before the worktree transfer

Risk

Defaults are unchanged. Cancellation is cooperative between generated tokens; prompt prefill and each individual token evaluation remain atomic.

@ivan-digital
ivan-digital merged commit 7b2d17f into main Aug 22, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant